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1.  Introduction 


1.1  Background 

During  the  combat  operations  phase  of  Operation  Iraqi  Freedom  (OIF) — March- April  2003,  U.S. 
Army  Patriot  air  and  missile  defense  (AMD)  units  were  involved  in  two  fratricide  incidents.  In 
the  first,  a  British  Tornado  was  misclassified  as  an  anti-radiation  missile  (ARM)  and 
subsequently  engaged  and  destroyed.  The  second  fratricide  incident  involved  a  Navy  F/A-18 
that  was  misclassified  as  a  tactical  ballistic  missile  (TBM)  and  also  engaged  and  destroyed. 

Three  flight  crew  members  lost  their  lives  in  these  incidents.  OIF  involved  a  total  of  1 1  Patriot 
engagements  by  U.S.  units.  Of  these  11,  nine  resulted  in  successful  TBM  engagements;  the 
other  two  (18%)  were  fratricides. 

In  the  spring  of  2004,  a  team  from  the  Army  Research  Laboratory’s  Human  Research  and 
Engineering  Directorate  (ARL  HRED)  began  looking  into  Patriot  performance  and  training 
issues  at  the  invitation  of  the  then  Ft.  Bliss  Commander,  Major  General  (MG)  Michael  A.  Vane. 
After  reviewing  the  conclusions  of  the  various  boards  of  inquiry  formed  to  look  into  the  OIF 
fratricides,  MG  Vane  was  convinced  that  human  performance  issues  were  part  of  the  problem 
leading  to  those  incidents.  He  was  particularly  concerned  by  what  he  tenned  a  “lack  of 
vigilance”  on  the  part  of  Patriot  operators  along  with  an  apparent  “lack  of  cognizance”  of  what 
was  being  presented  to  them  on  situation  displays  with  a  resulting  “absolute  trust  in  automation.” 
Accordingly,  he  requested  that  HRED  conduct  a  human-performance-oriented  critical  incident 
assessment  to  complement  the  board  of  inquiry  investigations  and  report  back  to  him  regarding 
potential  problems  and  solutions. 

HRED’s  project  team  spent  most  of  the  summer  and  fall  of  2004  performing  the  requested 
human-performance-oriented  critical  incident  assessment  of  the  OIF  fratricides.  An  initial 
assessment  was  delivered  to  MG  Vane  in  October  2004.  HRED’s  report  to  MG  Vane 
recommended  two  primary  actionable  items  to  redress  the  performance  problems  identified 
during  the  Patriot  Vigilance  effort: 

1 .  Re-examine  automation  concepts,  operator  roles,  and  command  and  control  (C2) 
relationships  in  AMD  battle  command  systems  to  emphasize  effective  human  supervisory 
control;  and 

2.  Develop  more  effective  missile  crews  and  C2  teams,  particularly  with  respect  to  Air 
Defense  Operations — re-look  the  level  of  expertise  required  to  operate  such  a  lethal  system 
on  the  modem  battlefield. 
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In  present  usage,  the  term  effective  human  supervisory  control  refers  to  a  situation  in  which 
Soldiers  and  not  the  automated  system  are  the  ultimate  decision  makers  in  AMD  firing  decisions. 
Uncritical  acquiescence  to  the  automated  system’s  recommendations  is  not  effective  human 
supervisory  control. 

In  a  report  on  Patriot  system  performance  requested  by  the  Under  Secretary  of  Defense  for 
Acquisition,  Technology,  and  Logistics,  the  Defense  Science  Board  reinforced  HRED’s 
recommendations  with  the  following  comments  (Defense  Science  Board,  2004).  Although  the 
full  report  is  classified,  these  extracts  are  not. 

The  Patriot  system  should  migrate  to  more  of  a  “man-in-the-loop”  philosophy 
versus  a  fully  automated  philosophy — providing  operator  awareness  and  control 
of  engagement  processes. 

and 


Patriot  training  and  simulations  should  be  upgraded  to  support  this  man-in-the- 
loop  protocol  including  the  ability  to  train  on  confusing  and  complex  scenarios 
that  contain  unbriefed  surprises. 

A  summary  of  the  report  on  Patriot  system  performance  is  available  for  download  on  the 
Defense  Science  Board  web  site.  Readers  should  note  that  the  report’s  recommendations  closely 
parallel  HRED’s  actionable  items,  and  both  address  the  need  for  AMD  training  modifications. 

In  addition  to  the  briefing  to  MG  Vane,  the  initial  phase  of  the  Patriot  Vigilance  project  also 
resulted  in  an  ARL  technical  report  titled  The  Human  Side  of  Automation:  Lessons  for  Air- 
Defense  Command  and  Control  (Hawley,  Mares,  &  Giammanco,  2005).  After  reviewing  results 
from  the  first  phase  of  the  project,  the  TRADOC  Capability  Manager  (TCM)  for  Lower  Tier 
AMD  systems  requested  that  the  Patriot  Vigilance  project  continue  into  a  second  year.  The 
TCM  specifically  requested  that  HRED’s  project  staff  expand  on  the  material  presented  in 
Hawley,  Mares,  and  Giammanco  (2005)  and  prepare  two,  more-detailed  reports,  one  concerned 
with  design  for  effective  human  supervisory  control  and  a  second  addressing  training  for  the 
emerging  class  of  AMD  systems.  The  intent  of  these  reports  was  to  infonn  the  AMD  community 
on  “what  right  looks  like”  in  each  of  these  topic  areas.  The  results  were  the  two  reports: 
Developing  Effective  Human  Supervisory  Control  for  Air  and  Missile  Defense  Systems  (Hawley 
&  Mares,  2006),  and  Training  for  Effective  Human  Supervisory  Control  of  Air  and  Missile 
Defense  Systems  (Hawley,  Mares,  &  Giammanco,  2006). 

In  the  late  summer  of  2005  after  MG  Vane  had  left  Ft.  Bliss  for  another  assignment,  the  project 
staff  briefed  his  replacement,  MG  (then  Brigadier  General)  Robert  P.  Lennox,  on  the  status  and 
results  of  the  Patriot  Vigilance  project.  MG  Lennox  formally  requested  that  the  project  be 
continued  for  at  least  another  year  so  that  the  technical  staff  could  work  with  the  AMD 
community  on  implementing  selected  results.  HRED’s  project  staff  also  would  participate  as  the 
MANPRINT  (Manpower  and  Personnel  Integration)  evaluator  during  a  Limited  User  Test  of  the 
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Post-Deployment  Build  6  (PDB-6)  software  suite  for  the  Patriot  system.  PDB-6  was  developed 
to  address  many  of  the  Patriot  system’s  operational  deficiencies  that  had  surfaced  during  OIF  and 
were  generally  considered  to  have  contributed  to  the  unacceptable  fratricide  rate. 

From  the  Fall  of  2005  through  the  Summer  of  2006  during  the  New  Equipment  Training  (NET) 
and  unit  train-up  period  for  the  PDB-6  Limited  User  Test,  the  FIRED  project  staffs  observations 
regarding  the  progress  of  training  for  the  test  unit  sounded  an  alarm  bell  loudly.  PDB-6  training 
was  not  progressing  as  it  should  have.  Training  events  were  being  completed,  but  individual  and 
crew  performance  objectives  were  not  being  met.  Many  of  the  training  issues  identified  and 
discussed  in  Hawley,  Mares,  and  Giammanco  (2006)  were  surfacing  and  were  not  being 
addressed  adequately  by  the  NET  process  or  follow-on  collective  training  by  the  test  unit. 
Moreover,  inadequate  unit  training  showed  up  clearly  in  the  ensuing  test  results. 

The  Army  Evaluation  Center’s  System  Assessment  Report  resulting  from  the  PDB-6  Limited 
User  Test  concluded  that  the  level  of  expertise  required  to  employ  the  Patriot  system  properly 
with  PDB-6  software  had  exceeded  the  current  Army  training  standard.  An  earlier  Army  board 
of  inquiry  report  on  the  OIF  fratricides  had  reached  a  similar  conclusion,  noting  that  “the  system 
[Patriot]  is  too  lethal  to  be  placed  in  the  hands  of  crews  trained  to  such  a  limited  standard.”  This 
convergence  of  evidence  supported  the  conclusion  that  modifications  to  current  AMD  training 
practices  were  required.  Moreover,  these  modifications  would  require  not  simply  more 
“traditional”  training,  but  performance-oriented  strategies  focused  on  deliberate  practice  (Hawley 
&  Mares,  2007).  Deliberate  practice  denotes  a  hands-on  instructional  regimen  focused  on 
specific  instructional  objectives  accompanied  by  immediate  and  expert  feedback  (Ericsson  & 
Charness,  1994).  Frequent  deliberate  practice  has  been  found  to  satisfy  a  primary  condition  for 
the  development  of  high  levels  of  job  expertise  (Ericsson,  Krampe,  &  Tesch-Romer,  1993). 

1.2  The  Reconfigurable  Tactical  Operations  simulator  (RTOS)  Operational 
Demonstration 

After  reviewing  results  from  the  OIF  boards  of  inquiry,  OIF  training-related  lessons  learned, 
conclusions  from  HRED’s  Patriot  Vigilance  project,  and  results  from  the  PDB-6  Limited  User 
Test,  the  Army  Air  Defense  Artillery  (ADA)  SchooFs  Directorate  of  Training,  Doctrine,  and 
Leader  Development  (DOTD-LD)  concurred  that  a  reexamination  of  AMD  training  practices  and 
strategies  was  required.  In  addition  to  general  agreement  that  a  change  in  training  rigor  and 
instructional  strategies  was  necessary,  DOTD-LD  identified  a  further  training  capability  gap 
across  AMD  forces.  This  gap  concerned  the  organic  simulation  capability  available  to  AMD 
units.  Their  conclusion  was  that  units  required  a  capability  to  train  fire  control  crews  across  the 
entire  AMD  kill  chain  that: 

1.  Supplemented  units’  organic  embedded  training  (ET)  capability; 

2.  Did  not  require  use  of  tactical  equipment;  and 
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3.  Could  be  conducted  in  an  environment  conducive  to  teambuilding,  coaching,  teaching,  and 
mentoring.  That  is,  a  capability  that  should  be  used  to  support  a  perfonnance-oriented 
instructional  strategy  focused  on  deliberate  practice. 

The  School’s  interest  in  an  enhanced  simulation  capability  to  supplement  ET  on  tactical 
equipment  was  initiated  by  an  Operational  Needs  Statement  produced  by  the  35th  ADA  Brigade 
in  Korea. 

The  ADA  School’s  concern  for  an  enhanced  unit  training  capability  was  heightened  by  that  fact 
that  upcoming  Base  Realignment  and  Closure  (BRAC)  moves  of  AMD  units  away  from  Ft.  Bliss 
will  amplify  the  training  capability  gap  by  denying  units  ready  access  to  Ft.  Bliss’  institutional 
(such  as  the  Patriot  Conduct  of  Fire  Trainers,  or  PCOFTs)  and  post-wide  training  resources  (such 
as  the  Drive  Up  Simulation  and  Training  facility). 

DOTD-FD  identified  an  existing  device,  the  Recon figurable  Tactical  Operations  Simulator 
(RTOS),  as  potentially  fulfilling  the  need  for  a  simulation  capability  to  supplement  units’  organic 
ET  capability,  while  not  requiring  the  use  of  tactical  equipment.  RTOS  hardware  is  commercial 
off  the  shelf  (COTS),  with  the  software  being  owned  by  the  Army.  The  device  has  been  in  use 
since  the  late  1970s  to  support  major  air  defense  exercises  as  well  as  higher  level 
experimentation  and  analysis.  However,  prior  to  the  current  project,  the  RTOS  had  not  been 
used  as  a  training  device. 

As  a  trainer,  RTOS  is  an  exemplar  of  a  medium-  to  high-fidelity,  part-task  training  device  for 
AMD  air  battle  operations.  The  term  medium-to  high-fidelity  means  the  RTOS  is  not  an  exact 
physical  replica  of  the  Patriot  system,  and  that  its  functional  characteristics  are  not  identical  to 
those  of  the  tactical  system.  Table  1  presents  and  defines  a  set  of  terms  commonly  used  to 
describe  simulators  and  simulations.  Part-task  means  that  the  RTOS  does  not  support  all  AMD 
air  battle  operations  tasks.  The  term  reconfigurable  means  that  the  RTOS’  physical  features 
(e.g.,  displays  and  controls)  are  not  hard-manufactured  or  hard-wired  into  the  device.  Rather, 
displays  are  software  driven  and  controls  are  implemented  either  using  a  mouse  or  a  touch- 
sensitive  screen.  The  original  intent  behind  the  RTOS  was  that  any  potential  AMD  system  or 
system  modification  could  be  mocked-up  readily  and  used  in  demonstrations  or  experiments.  A 
more  complete  description  of  the  RTOS  and  its  capabilities  is  provided  in  appendix  A. 

Fess-than-full-fidelity  training  devices  such  as  the  RTOS  are  commonly  used  in  aviation  and  in 
other  training  applications  (see,  for  example,  Johnson  &  Stewart,  2005).  Because  of  their 
relatively  low  cost  versus  full-fidelity  training  equipment,  devices  such  as  the  RTOS  have  been 
found  to  be  particularly  useful  in  supporting  essential  skills  integration  (e.g.,  learning  decision 
processes  that  underlie  weapons  system  use)  before  moving  to  more  costly  and  limited  full- 
fidelity  training  settings  (Alexander,  Brunye,  Sidman,  &  Weil,  2005).  Other  research  discussed 
in  Stewart,  Johnson,  and  Howse  (2007)  suggests  that  less-than-full-fidelity  training  devices 
might  even  be  beneficial  for  novice  to  mid-level  trainees.  Novice  and  mid-level  trainees  who  are 
still  making  mistakes  and  learning  through  feedback  may  benefit  from  a  simpler,  more  focused 
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training  environment,  more  suitable  for  training  primary  skills.  For  trainees  at  the  early-  to  mid¬ 
level  of  learning,  high  fidelity  may  provide  no  advantage  for  transfer  to  the  target  performance 
setting,  and  may  actually  introduce  too  much  complexity  into  an  already  novel  and  stressing 
environment.  In  other  words,  the  sheer  complexity  of  the  actual  equipment  setting  in  AMD  may 
impede  training  for  novice  to  mid-level  trainees  by  making  it  more  difficult  to  concentrate  on  the 
tasks  at  hand. 


Table  1 .  Commonly  used  simulation  terms  and  definitions. 


Physical  fidelity 

The  degree  to  which  the  physical  simulation  looks,  sounds,  and  feels  like  the  operational 
environment  in  terms  of  visual  displays,  controls,  and  audio  as  well  as  the  physical 
models  driving  each  of  these  variables. 

Functional  fidelity 

The  degree  to  which  the  simulation  acts  like  the  operational  environment  in  reacting  to 
the  tasks  executed  by  the  trainee. 

Buy-In 

The  degree  to  which  a  person  recognizes  that  an  event  or  experience  is  useful  for 
training.  Acceptance  of  the  simulation  as  “relevant,”  and  a  willingness  to  use  the 
environment  to  practice  the  behaviors  targeted  by  the  training  program. 

Source:  Alexander,  Brunye,  Sidman,  &  Weil  (2005) 


1.3  Demonstration  Objectives 

The  RTOS  Operational  Demonstration  (OpDemo)  was  organized  as  a  joint  project  involving  the 
ADA  School’s  DOTD-LD  and  the  5th  Battalion  52nd  ADA  Regiment  (5-52  ADA).  5-52  ADA  is 
part  of  the  1 1th  ADA  Brigade.  Personnel  from  ARL’s  HRED  participated  with  DOTD-LD  and 
5-52  ADA  in  a  technical  advisory  capacity.  Technical  support  to  the  RTOS  and  other  technical 
aspects  of  the  OpDemo  was  provided  by  the  simulator’s  developer,  Science  Applications 
International  Corporation  (SAIC). 

DOTD-LD’s  objectives  in  the  RTOS  OpDemo  were  to: 

1 .  Assess  whether  the  RTOS  could  supplement  existing  training  in  the  areas  of  (a)  problem¬ 
solving  for  reaction  to  off-nominal  engagement  situations  and  (b)  advanced  Joint  Air 
Defense  Operations. 

2.  Assess  the  utility  of  the  RTOS  for  unit-level  crew  training  in  air  defense  operations  in 
preparation  for  PDB-6  New  Equipment  Training. 

5-52  ADA’s  objective  in  the  OpDemo  was  to  train  fire  control  crews  in  an  environment 
conducive  to  coaching,  teaching,  and  mentoring  using  RTOS  consoles  and  multiple  engagement 
scenarios.  Their  interest  in  the  OpDemo  was  motivated  by  a  desire  to  conduct  additional  unit- 
level  crew  training  in  preparation  for  PDB-6  NET. 

In  essence,  the  OpDemo  was  focused  on  evaluating  the  feasibility,  utility,  and  potential 
effectiveness  of  a  new  approach  to  AMD  unit  training  emphasizing  deliberate  practice  and  using 
the  RTOS  as  a  training  vehicle.  This  is  an  important  point  and  bears  repeating.  The  OpDemo 
was  not  simply  about  the  RTOS.  It  was  focused  on  two  separate  but  related  objectives:  (1)  the 
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demonstration  and  evaluation  of  modified  instructional  methods  for  use  in  AMD  unit  training, 
and  (2)  an  assessment  of  the  potential  utility  of  the  RTOS  as  a  device  to  support  AMD  unit 
training. 


2.  Method 


2.1  Participants 

Forty-two  Soldiers  from  5-52  ADA  participated  in  the  OpDemo  during  the  period  16-27  April 
2007.  Time  in  service  for  the  participants  ranged  from  18.5  years  to  one  year,  with  a  median 
length  of  service  of  two  years.  In  terms  of  time  in  their  current  job  position,  participant 
responses  ranged  from  a  high  of  seven  years  to  a  low  zero  (right  out  of  Advanced  Individual 
Training  [AIT]).  The  median  time  in  their  current  job  position  was  1 1.5  months. 

Personnel  from  5-52’s  Alpha  (A),  Bravo  (B),  Charlie  (C),  and  Delta  (D)  batteries  formed  the  fire 
unit  crews  used  in  the  OpDemo — Engagement  Control  Station  (ECS)  Tactical  Control  Officers 
(TCOs)  and  Tactical  Control  Assistants  (TCAs).  Personnel  from  5-52’s  Headquarters  and 
Headquarters  Battery  (HHB)  formed  the  Information  and  Coordination  Central  (ICC)  crews — 
Tactical  Director  (TD)  and  Tactical  Director  Assistant  (TDA).  Instructor  personnel  used  in  the 
OpDemo  were  from  5-52’s  EMMO  (Electronic  Missile  Maintenance  Officer)  team  and  the  ADA 
School’s  DOTD-LD.  In  Patriot  units,  battalion  EMMO  personnel  provide  training  and 
evaluation  support  to  component  batteries. 

2.2  RTOS  Layout 

The  RTOS  stations  used  in  the  OpDemo  were  set  up  as  a  suite  of  eight  consoles  representing 
three  Patriot  fire  unit  (FU)  ECS  stations  and  one  ICC.  An  additional  workstation  representing  an 
external  Patriot  battalion  was  used  as  the  Air  Defense  Artillery  Fire  Control  Officer  (ADAFCO) 
display.  Additionally,  a  simulation  and  support  workstation  area  was  set  up  to  allow  the 
instructors  to  both  create  and  distribute  scenarios.  Figure  1  proves  a  graphic  depiction  of  the 
RTOS  layout  used  in  the  Demonstration.  The  site  for  the  OpDemo  was  5-52  ADA’s  Tactical 
Fire  Direction  Facility  located  at  the  Tobin  Wells  range  area  on  Ft.  Bliss. 

2.3  Air  Battle  Operations  Tasks 

Twenty-three  AMD  air  battle  operations  tasks  were  chosen  for  evaluation  during  the  OpDemo. 
All  of  these  are  critical  Patriot  operator  tasks  for  Military  Occupational  Specialties  (MOSs)  14E 
(enlisted)  and  14E/140E  (officer/warrant  officer).  In  addition,  four  high-priority  operator 
activities  associated  with  specific  scenarios  used  during  the  OpDemo  but  not  formally  defined  as 
tasks  were  added  to  the  task  list.  The  27  tasks  included  in  the  OpDemo  (23  “standard”  plus  four 
special)  are  listed  in  tables  2a  and  2b.  In  tables  2a  and  2b,  tasks  are  categorized  by  job  position: 
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TCA/TDA  or  TCO/TD.  Tasks  24-27  are  the  extra  activities  included  in  the  OpDemo.  These 
tasks  pertain  to  ARM  classification  procedures  and  FEZ-JEZ-MEZ  operations.  (Note:  FEZ 
refers  to  a  Fighter  Engagement  Zone;  JEZ  refers  to  a  Joint  Engagement  Zone;  and  MEZ  refers  to 
a  Missile  Engagement  Zone.)  ARM  classification  procedures  were  selected  for  inclusion  in  the 
OpDemo  because  they  represent  an  off-nominal  performance  situation  that  contributed  to  the 
Tornado  fratricide  during  the  combat  operations  phase  of  OIF  (see  Hawley  &  Mares,  2006). 
FEZ-JEZ-MEZ  procedures  are  an  important  aspect  of  advanced  Joint  Air  Defense  Operations. 
The  FEZ-JEZ-MEZ  exercise  permitted  the  trainees  to  participate  in  the  complete  kill  chain  for 
Joint  Air  Defense  Operations  from  the  ADAFCO  located  at  the  Air  Force  Controlling  Authority 
down  through  the  Patriot  ICC  to  the  ECS.  Other  unit-level  training  capabilities  such  as  the 
embedded  trainers  in  Patriot  tactical  shelters  or  the  PCOFT  at  the  ADA  School  do  not  currently 
support  training  on  joint  kill  chain  operations. 


[FU  1  ] 


[FU3] 


[ICC] 


1 


[ADAFCO  ] 


Figure  1.  Reconfigurable  Tactical  Operations  Simulator  (RTOS)  layout  for  the  operational  demonstration. 
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2.4  Training  Method  and  Instructional  Approach 

Beyond  the  potential  use  of  the  RTOS  in  a  unit  setting,  the  OpDemo  was  intended  to  illustrate 
several  modified  approaches  to  AMD  air  battle  operations  training  advocated  in  Hawley  and 
Mares  (2007).  The  first  of  these  involved  using  the  simulator  in  an  open  instructional 
environment  conducive  to  coaching  and  mentoring.  As  trainees  went  through  the  various  RAL 
(Reticule  Aim  Level — a  proxy  for  scenario  difficulty)  scenarios  used  in  the  OpDemo,  instructor 
cadre  were  immediately  available  to  answer  questions  and  provide  corrective  guidance.  Each 
scenario  was  followed  by  a  scenario-specific  after  action  review  (AAR)  during  which  the 
trainees’  performance  was  critiqued.  It  was  also  possible  to  stop  the  scenario  at  any  point  to 
review  an  important  concept  or  critique  and  correct  participant  perfonnance  to  that  point.  The 
stop-action  feature  was  used  several  times  during  the  OpDemo. 

Several  segments  of  the  OpDemo  also  were  used  explicitly  to  illustrate  and  explore  the  benefits 
of  a  deliberate  practice  approach  to  training  selected  air  battle  operations  tasks.  As  noted 
previously,  deliberate  practice  denotes  a  hands-on  instructional  regimen  focused  on  specific 
instructional  objectives  accompanied  by  immediate  and  expert  feedback  (Ericsson  &  Charness, 
1994).  The  deliberate  practice  portion  of  the  OpDemo  was  implemented  using  what  might  be 
tenned  a  “chunk  and  simulate”  fonnat.  Trainees  were  first  provided  with  a  brief  (15-20  minute) 
slide  presentation  addressing  the  specific  instructional  objective  at  hand  (the  “chunk”  portion). 
This  was  followed  by  a  relatively  short,  simulation-based  practical  exercise  (30  minutes) 
focusing  on  the  concepts  covered  in  the  presentation.  Trainees  were  thus  immediately  allowed  to 
practice  the  concepts  covered  in  the  up-front  presentation.  The  simulation  portion  of  the  exercise 
was  followed  by  a  focused  AAR  that  reviewed  the  concept  under  consideration,  critiqued  the 
trainees’  performance,  and  provided  corrective  guidance.  Deliberate  practice  exercises  were 
developed  by  DOTD-LD  for  ARM  classification  procedures  and  FEZ-JEZ-MEZ  operations. 


Table  2a.  Tactical  Control  Assistant  (TCA)  and  Tactical  Director  Assistant  (TDA)  tasks  used  in  the 
operational  demonstration. 


TCA/TDA  Tasks 

1 

441-083-1478  Engage  Targets  on  the  Engagement  Control  Station  (ECS) 

2 

441-083-1479  Evaluate  Preengagement  Data  at  the  Engagement  Control  Station 

3 

441-083-1480  Evaluate  Preengagement  Data  at  the  Information  and  Coordination  Central 

4 

441-083-1482  Initiate  Target  Engagement  at  the  Information  and  Coordination  Central 

5 

441-083-1486  Perform  Friendly  Protect  at  the  Engagement  Control  Station  (ECS) 

6 

441-083-1487  Perform  Friendly  Protect  on  the  Information  and  Coordination  Central  (ICC) 

7 

441-084-1407  Perform  Engagement  Control  Station  (ECS)  Initialization 

8 


Table  2b.  Tactical  Control  Officer  (TCO)  and  Tactical  Director  (TD)  tasks  used  in  the  operational  demonstration. 


TCO/TD  Tasks 

8 

441-EW1-1084  Perform  Engagement  Control  Station  (ECS)  Initialization 

9 

441-EW1-1085  Perform  Engagement  Control  Station  (ECS)  Reinitialization 

10 

441-EW1-1087  Direct  Firing  Battery  System  Reorientation 

11 

441-EW1-1088  Perform  Tactical  Ballistic  Missile  (TBM)  Engagement  Operations 

12 

441-EW1-1089  Perform  Air  Defense  Mission  in  all  Modes  of  Control  for  Engagement  Control  Station 
(ECS) 

13 

441-EW1-1090  Identify  Targets  at  Engagement  Control  Station  (ECS) 

14 

44 1  -EW 1-1091  Supervise  Engagement  of  T argets 

15 

441-EW 1-1093  Implement  Firing  Doctrine  Changes 

16 

441-EW1-1094  Define  Engagement  Control  Station  (ECS)  Tabular  Displays 

17 

441-EW1-1095  Implement  Air  Breathing  Threat  (ABT)  and  Tactical  Ballistic  Missile  (TBM)  Search  using 
Expanded  Search  Sectors 

18 

441-EW1-1099  Supervise  the  Process  of  Airspace  Control  Orders  (ACO) 

19 

44 1  -EW  1-1113  Supervise  Engagement  Operations 

20 

441-EW1-1 121  Analyze  the  Process  of  Evaluation,  Decision,  and  Weapons  Assignment  (EDWA) 

21 

441-EW1-1122  Perform  Friendly  Protection 

22 

441-EW1-1123  Perform  Duties  and  Responsibilities  as  the  Tactical  Control  Officer  (TCO)  and  Tactical 
Control  Assistant  (TCA) 

23 

441-EW1-1124  Perform  Air  Breathing  Threat  (ABT)  and  Tactical  Ballistic  Missile  (TBM)  Search  Using 
Expanded  Search  Sectors 

24 

Identify  Mis-Classified  Tracks  that  are  displayed  as  ARM,  TBM,  or  ABT  symbology 

25 

Identify  when  crew  duties  and  responsibility  should  be  modified 

26 

Use  of  a  banded  FEZ,JEZ,  MEZ  TTPs 

27 

Performing  De-Lousing  TTPs 

2.5  Procedure 

An  overview  of  the  structure  of  the  OpDemo  is  provided  in  table  3.  Each  of  5-52  ADA’s 
batteries  (A,  B,  C,  and  D)  received  a  full  day  of  training  each  week,  for  a  total  of  two  days.  The 
Fridays  of  each  week  were  shared  by  the  batteries  and  were  focused  on  battalion-specified 
training  activities.  Tables  4  and  5  provide  a  break-out  of  OpDemo  activities  for  each  day.  Day  1 
refers  to  each  battery’s  training  day  during  the  first  week;  Day  2  refers  to  that  battery’s  training 
day  during  the  second  week.  Table  5  shows  the  planned  break-out  of  activities  for  the  Fridays 
shared  by  the  batteries,  denoted  as  Day  3. 

Day  1  began  with  a  presentation  that  provided  an  overview  of  the  RTOS  and  discussed  the  intent 
of  the  OpDemo.  Progressive  RAL  scenarios  then  commenced.  Each  scenario  was  followed  by  a 
Hotwash  during  which  each  of  the  fire  unit  crews,  the  ICC  crew,  the  EMMO  team,  and  anyone 
else  participating  in  the  exercise  (e.g.,  the  simulated  ADAFCO)  was  pennitted  to  express  their 
opinions  both  pro  and  con  on  what  had  transpired.  The  DOTD-developed  FEZ-JEZ-MEZ 
exercise  took  place  during  the  afternoon  of  Day  1. 

Day  2  opened  with  a  written  pretest  addressing  the  knowledge  component  of  the  ARM 
Classification  exercise.  The  pretest  was  followed  by  the  ARM  Classification  chunk  and  simulate 
training  module.  Following  the  training  module,  the  ARM  Classification  knowledge  test  was 
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administered  again  as  a  posttest.  The  ARM  Classification  knowledge  test  used  in  the  OpDemo  is 
shown  in  appendix  B.  At  the  end  of  Day  2,  demonstration  participants  were  asked  to  complete 
two  surveys.  The  first  survey  asked  them  to  evaluate  the  RTOS  in  terms  of  how  well  it 
supported  the  27  air  battle  operations  tasks  included  in  the  OpDemo.  Participants  were  asked  to 
provide  one  evaluation  for  each  task,  with  three  possible  levels  of  evaluation:  Does  Not  Support 
(N/S),  Partially  Supports  (P/S),  or  Fully  Supports  (F/S).  The  first  survey  addresses  what  Bell 
and  Waag  (1998)  and  Johnson  and  Stewart  (2005)  refer  to  as  the  simulator’s  utility  for  training 
the  tasks  under  consideration.  The  Utility  survey  is  shown  in  appendix  C. 

The  second  survey  administered  at  the  end  of  Day  2  addressed  the  issue  of  trainee  Buy-In 
(reference  table  1);  it  also  pennitted  the  participants  to  provide  open-ended,  written  comments  on 
the  OpDemo,  the  instructional  procedures  employed,  or  the  RTOS  as  a  training  device,  along 
with  suggestions  for  improvements  in  any  of  those  categories.  This  survey,  denoted  the  RTOS 
Demo  Questionnaire,  is  shown  in  appendix  D. 

Although  the  OpDemo  began  with  the  schedule  of  activities  outlined  in  tables  4  and  5,  deviations 
from  the  plan  occurred.  For  example,  some  of  the  battery  crews  were  more  advanced  than  others 
and  progressed  more  rapidly,  thus  requiring  the  use  of  more  difficult  RAL  scenarios  near  the  end 
of  their  scheduled  time.  Similarly,  some  of  the  battery  crews  were  not  as  far  along  in  their 
training  sequence  as  had  been  assumed,  and  thus  required  more  practice  on  less  complex 
scenarios.  The  schedule  was  also  adjusted  to  accommodate  visits  to  the  OpDemo  site  by  VIPs. 
These  schedule  adjustments  permitted  the  VIPs  to  view  and  comment  on  salient  aspects  of  the 
OpDemo  such  as  the  ARM  Classification  and  FEZ-JEZ-MEZ  training  modules.  VIPs  visiting 
the  OpDemo  site  included  the  Ft.  Bliss  Commanding  General  and  members  of  his  staff,  the 
Commander  of  the  32nd  Army  AMD  Command  (AAMDC),  various  AMD  brigade  and  battalion 
commanders  and  their  staffs,  and  representatives  from  the  AMD  Lower  Tier  Project  Office. 
Patriot  is  classified  as  an  AMD  Lower  Tier  system. 


Table  3.  Operational  demonstration  training  support  timeline:  16  to  27  April  2007. 


April 

MON 

TUE 

WED 

THUR 

FRI 

16  ALPHA  (A) 

17  BRAVO  (B) 

18  CHARLIE  (C) 

19  DELTA  (D) 

20  BRAVO  (Two 
Crews) 

CHARLIE  (One 
Crew) 

23  ALPHA 

24  BRAVO 

25  CHARLIE 

26  DELTA 

27  ALPHA  (One 
Crew) 

DELTA  (Two 
Crews) 
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Table  4.  Daily  training  schedule  for  days  1  and  2. 


DAY  1 

DAY  2 

0900-0930  RTOS  Overview 

0900-0930  Written  Test 

0930-0950  Crew  in  brief 

0930-0950  ARM  Presentation 

0950-1000  Break 

0950-1000  Break 

1000-1045  1st  Air  Battle  (RAL  5) 

1000-1045  ARM  Scenario  with  discussion 

1045-1050  Break 

1045-1100  Written  Test 

1050-1105  Hotwash 

1100-1115  Break 

1 105-1150  2nd  Air  Battle  (RAL  9) 

1 1 15-1200  Test  review  and  Hot  Wash 

1150-1200  Hotwash 

1200-1300  Lunch 

1200-1300  Lunch 

1300-1315  Crew  in  brief 

1300-1330  Crew  in  brief  (DOTD  JEZ) 

1315-1350  5th  Air  Battle 

1330-1400  3rd  Air  Battle  (DOTD  JEZ) 

1350-1400  Break 

1400-1415  Break 

1400-1430  Hotwash 

1415-1450  Hotwash  with  discussion 

1430-1440  Break 

1450-1500  Break 

1440-1500  Crew  in  brief 

1500-1550  4th  Air  Battle  (RAL  11) 

1500-1550  6th  Air  Battle 

1550-1600  Break 

1550-1600  Break 

1600-1630  AAR  (for  both  scenario  and  day) 

1600-1630  AAR/  Written  test  (for  both  scenario  and  day) 

Table  5.  Daily  training  schedule  for  day  3. 


DAY  3 


0900-0915 

0915-0950 

0950-1000 

1000-1030 

1030-1115 

1115-1130 

1130-1200 

1200-1300 

1300-1315 

1315-1350 

1350-1400 

1400-1430 

1440-1600 

1600-1630 


Crew  in  brief 

8th  Air  Battle  (BN  Focus) 

Break 

Hotwash 

9th  Air  Battle  (BN  Focus) 

Break 

Flotwash 

Lunch 

Crew  in  brief 

10th  Air  Battle  (BN  Focus) 

Break 

Flotwash 

Unit  Focused  Events 
AAR/  RTOS  Surveys 
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3.  Results 


3.1  Utility  Evaluation 

Data  from  the  Utility  assessment  of  air  battle  operations  tasks  are  presented  in  tables  6a  and  6b. 
Table  6a  presents  results  for  TCA  and  TDA  tasks,  while  table  6b  presents  results  for  TCO  and 
TD  tasks.  Responses  to  the  Utility  survey  were  first  screened  to  ensure  that  respondents  held  the 
proper  MOS.  This  initial  screening  resulted  in  five  participants  being  eliminated  from  the 
analysis  (all  25Fs  or  14Js).  The  remaining  surveys  were  then  screened  again  to  ensure  that 
respondents  had  the  proper  background  (job  assignment,  time  in  the  job  position,  etc.)  to  provide 
a  valid  assessment  of  the  RTOS’  utility  for  training  the  task  in  question.  Screenings  were  done 
by  Patriot  subject  matter  experts  (SMEs)  from  DOTD-LD.  The  secondary  screening  resulted  in 
the  numbers  given  in  the  table  columns  labeled  Total. 

The  utility  assessment  results  shown  in  tables  6a  and  6b  indicate  that  a  large  majority  of  the 
participants  judged  the  RTOS  to  have  utility  for  training  the  air  defense  operations  tasks  used  in 
the  OpDemo.  In  most  cases,  a  large  majority  of  the  survey  responses  were  for  fully  supports 
(F/S)  or  partially  supports  (P/S).  The  only  exceptions  to  this  general  observation  are  for  Tasks  7, 
8  and  9.  These  tasks  all  concern  System  Initialization  and  Reinitialization.  At  present,  the 
RTOS  does  not  support  these  tasks  fully. 

After  reviewing  results  from  the  Utility  survey,  Patriot  SMEs  from  DOTD-LD  cautioned  that 
because  of  their  lack  of  experience  some  participants  might  not  have  fully  understood  the 
complete  scope  of  the  tasks  involved  or  what  the  survey  questions  implied.  Consequently,  while 
the  Utility  results  support  a  general  conclusion  that  the  RTOS  has  value  for  training  air  defense 
operations  tasks,  further  inquiry  is  necessary  before  making  definitive  conclusions  regarding  the 
device’s  specific  strengths  or  weaknesses.  Clearly,  however,  most  of  the  participants  came  away 
from  the  OpDemo  with  a  favorable  impression  of  the  RTOS  and  its  potential  use  as  a  unit-level 
training  device. 
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Table  6a.  Utility  evaluation  ratings  for  Tactical  Control  Assistant  (TCA)  and  Tactical  Director  Assistant  (TDA) 
tasks. 


TCA/TDA  Tasks 

Total 

N/S 

P/S 

F/S 

1 

Engage  Targets  on  the  Engagement  Control  Station  (ECS) 

20 

0 

5 

15 

2 

Evaluate  Preengagement  Data  at  the  Engagement  Control  Station 

21 

1 

4 

16 

3 

Evaluate  Preengagement  Data  at  the  Information  and  Coordination 
Central 

12 

0 

1 

11 

4 

Initiate  Target  Engagement  at  the  Information  and  Coordination 
Central 

12 

0 

1 

11 

5 

Perform  Friendly  Protect  at  the  Engagement  Control  Station  (ECS) 

22 

1 

5 

16 

6 

Perform  Friendly  Protect  on  the  Information  and  Coordination 

Central  (ICC) 

18 

1 

1 

16 

7 

Perform  Engagement  Control  Station  (ECS)  Initialization 

20 

3 

8 

9 

Table  6b.  Utility  evaluation  ratings  for  Tactical  control  Officer  (TCO)  and  Tactical  director  (TD)  tasks. 


TCO/TD  Tasks 

Total 

N/S 

P/S 

F/S 

8 

Perform  Engagement  Control  Station  (ECS)  Initialization 

17 

0 

6 

11 

9 

Perform  Engagement  Control  Station  (ECS)  Reinitialization 

17 

0 

9 

8 

10 

Direct  Firing  Battery  System  Reorientation 

19 

0 

6 

13 

11 

Perform  Tactical  Ballistic  Missile  (TBM)  Engagement  Operations 

17 

0 

0 

17 

12 

Perform  Air  Defense  Mission  in  all  Modes  of  Control  for 

Engagement  Control  Station  (ECS) 

17 

2 

4 

11 

13 

Identify  Targets  at  Engagement  Control  Station  (ECS) 

19 

0 

1 

18 

14 

Supervise  Engagement  of  Targets 

19 

0 

1 

18 

15 

Implement  Firing  Doctrine  Changes 

19 

0 

2 

17 

16 

Define  Engagement  Control  Station  (ECS)  Tabular  Displays 

19 

0 

1 

18 

17 

Implement  (ABT)  and  Tactical  Ballistic  Missile  (TBM)  Search  using 
Expanded  Search  Sectors 

19 

0 

2 

17 

18 

Supervise  the  Process  of  Airspace  Control  Orders  (ACO) 

19 

3 

2 

14 

19 

Supervise  Engagement  Operations 

19 

0 

3 

16 

20 

Analyze  the  Process  of  Evaluation,  Decision,  and  Weapons 
Assignment  (EDWA) 

19 

0 

4 

15 

21 

Perform  Friendly  Protection 

19 

0 

3 

16 

22 

Perform  Duties  and  Responsibilities  as  the  Tactical  Control  Officer 
(TCO) 

18 

0 

1 

17 

23 

Perform  Air  Breathing  Threat  (ABT)  and  Tactical  Ballistic  Missile 
(TBM)  Search  Using  Expanded  Search  Sectors 

19 

0 

1 

18 

24 

Identify  Mis-Classified  Tracks  that  are  displayed  as  ARM,  TBM,  or 
ABT  symbology 

19 

0 

1 

18 

25 

Identify  when  crew  duties  and  responsibility  should  be  modified 

19 

0 

2 

17 

26 

Use  of  a  banded  FEZ,JEZ,  MEZ  TTPs 

19 

0 

2 

17 

27 

Performing  De-Lousing  TTPs 

19 

0 

2 

17 

13 


3.2  Pre-  Post-Test  Results 


Bell  and  Waag  (1998)  list  three  categories  of  approach  for  evaluating  the  training  effectiveness 
of  simulators.  Utility  evaluations  are  the  easiest  and  quickest  to  obtain,  and  are  reported  in  the 
previous  subsection.  The  second  category  of  evaluation  is  in-simulator  learning.  Trainees 
practice  tasks  in  the  simulator  and  thereby  show  learning  through  an  improvement  in 
performance.  Typically,  the  method  is  one  of  pre-test,  practice,  and  then  post-test.  This  general 
approach  was  used  to  assess  the  training  effectiveness  of  the  RTOS  coupled  with  the  chunk-and- 
simulate  instructional  approach  for  the  ARM  Classification  module.  Bell  and  Waag’s  third 
category,  transfer  of  training  to  the  job  setting,  was  beyond  the  scope  of  the  OpDemo.  The 
project  staff  will,  however,  continue  to  track  the  transfer  of  training  issue  during  subsequent 
training  and  performance  certification  activities  involving  5-52  ADA. 

Pre-  and  post-training  results  for  the  ARM  Classification  module  are  summarized  in  table  7.  The 
mean  score  on  the  pre-training  knowledge  test  was  18.78.  After  the  chunk-and-simulate  session, 
the  mean  score  was  67.92.  The  mean  improvement  across  participants  was  49.14  points. 
Difference  scores  for  each  of  the  37  MOS-qualified  participants  completing  both  the  pre-  and 
post-tests  were  evaluated  using  a  paired-sample  t-test.  The  t-test  resulted  in  an  observed  t 
statistic  of  12.91  with  36  degrees  of  freedom  (p  <  .001).  The  differences  in  knowledge  test 
performance  pre-  and  post-training  module  are  large  and  statistically  significant. 

Table  7.  Pre-  and  post-training  results  for  the  ARM  classification 
training  module. 


Preliminary 

Test 

Post- 

Training 

Test 

Change  Score 

N=37 

Mean 

18.78 

67.92 

49.14 

SD 

16.73 

23.95 

23.15 

3.3  Buy-In  Results 

Recall  from  table  1  that  Buy-In  is  defined  as  “the  degree  to  which  a  person  recognizes  that  an 
event  or  experience  is  useful  for  training:  Acceptance  of  the  simulation  as  ‘relevant,’  and  a 
willingness  to  use  the  environment  to  practice  the  behaviors  targeted  by  the  training  program.” 
Buy-In  is  an  important  consideration  with  respect  to  the  selection  of  any  simulator.  Research  has 
shown  (e.g.,  Salas,  Bowers,  and  Rhodenizer,  1998)  that  trainee  buy-in,  or  acceptance  of  a 
simulator  or  instructional  setting  as  a  legitimate  training  experience,  is  an  important  determinant 
of  whether  they  will  willingly  use  it  and  take  the  training  experience  seriously. 

Buy-In  was  assessed  using  the  five  statements  listed  as  follows: 
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1.  The  RTOS  unit  training  environment  is  more  conducive  to  coaching,  teaching,  and 
mentoring  than  the  current  method  used  to  train. 

2.  I  learned  new  advanced  Joint  Air  Defense  Operations  techniques. 

3.  I  learned  techniques  to  evaluate  and  react  to  off-nominal  situations 
(misclassifications,  anomalies,  etc.). 

4.  All  things  considered,  this  demonstration  was  a  worthwhile  training  event. 

5.  I  think  the  RTOS  will  be  a  useful  tool  for  unit  training. 

Participants  were  asked  to  agree  or  disagree  with  each  statement. 

Response  totals  for  each  statement  are  given  in  table  8.  For  each  statement,  a  large  majority  of 
participants  indicated  agreement  with  the  statement.  For  statements  4  and  5,  participant 
agreement  was  unanimous  (100%  Agree):  Participants  uniformly  agreed  that  the  Demonstration 
was  a  worthwhile  training  event  and  that  the  RTOS  would  be  a  useful  tool  for  unit  training.  The 
lowest  percentage  of  favorable  response  was  for  statement  1.  Here,  only  37  of  the  42  (88.1%) 
participants  expressed  agreement  that  the  RTOS  as  used  in  the  OpDemo  was  more  conducive  to 
coaching,  teaching,  and  mentoring  than  the  current  method  used  to  train. 

Table  8.  Buy-In  survey  results. 


Statement 

Agree 

Pet  (%) 

Disagree 

Pet  (%) 

Total 

1 

37 

88.10% 

5 

11.90% 

42 

2 

40 

95.24% 

2 

4.74% 

42 

3 

40 

95.24% 

2 

4.76% 

42 

4 

42 

100.00% 

0 

0.00% 

42 

5 

42 

100.00% 

0 

0.00% 

42 

3.4  Open-Ended  Comments 

The  RTOS  Demo  Questionnaire  also  permitted  participants  to  provide  open-ended  comments 
concerning  their  experiences  during  the  OpDemo.  Two  open-ended  questions  were  posed  to 
OpDemo  participants: 

1.  Comments?  (Environment,  Training,  RTOS  Issues,  General  Impressions) 

2.  What  does  the  RTOS  need  to  do  that  it  does  not  currently  do? 

A  list  of  all  participant  responses  to  item  1  is  provided  in  appendix  E.  Participant  responses  to 
the  second  question  concerning  RTOS  limitations  generally  tended  to  reflect  the  same  issues  and 
opinions  obtained  during  the  post-scenario  Hotwash  sessions.  These  results  are  addressed  in  the 
next  subsection  on  Hotwash  comments.  For  each  comment  in  appendix  E,  the  respondent’s  rank 
and  current  job  position  or  MOS  are  provided  as  an  identifying  header. 


15 


A  review  of  participant  responses  in  appendix  E  indicated  an  overall  favorable  opinion  of  the 
OpDemo,  the  RTOS,  and  the  way  it  was  used — the  instructional  approach  focusing  on  deliberate 
practice.  Participant  comments  were  consistent  with  the  Utility  and  Buy-In  results  presented 
previously.  Many  participants  picked  up  on  and  commented  favorably  on  the  fact  that  the 
instructional  experience  provided  during  the  OpDemo  was  different  from  standard  training 
practices  in  AMD  units.  They  liked  the  way  training  was  done  during  the  OpDemo.  For  the 
several  respondents  remarking  that  they  liked  the  RTOS  better  than  the  PCOFT,  follow-on,  one- 
on-one  interviews  indicated  that  their  unfavorable  comments  regarding  the  PCOFT  resulted  from 
the  manner  in  which  the  PCOFT  is  currently  used  rather  than  any  negative  impressions  of  the 
simulator  itself. 

3.5  Hotwash  Comments 

At  the  conclusion  of  each  scenario  session  a  Hotwash  was  conducted.  In  present  usage,  the  tenn 
Hotwash  refers  to  a  debrief  session  during  which  each  group  of  participants  was  permitted  to 
comment  on  the  conduct  of  the  exercise,  their  experiences  with  the  RTOS,  or  any  other  topic 
they  wished  to  bring  up.  A  total  of  30  Hotwash  sessions  were  conducted  over  the  two  weeks  of 
the  OpDemo.  For  purposes  of  analysis,  positive  comments  and  criticisms  from  all  30  Hotwash 
sessions  were  aggregated  by  common  themes.  A  total  of  three  positive  and  four  negative  themes 
ran  through  participant  Hotwash  comments. 

The  three  positive  Hotwash  themes  that  emerged  from  participant  comments  were: 

1 .  Participants  liked  the  overall  training  approach  characterized  by  an  open  instructional 
environment  with  ready  access  to  instructor  cadre  for  feedback  and  corrective  guidance. 

2.  Participants  liked  the  chunk-and-simulate  instructional  approach  used  for  the  JEZ-FEZ- 
MEZ  and  ARM  Classification  training  modules. 

3.  Participants  viewed  the  OpDemo  as  an  enjoyable,  motivating  instructional  experience. 

With  respect  to  the  third  Hotwash  theme,  one  relatively  experienced  participant  remarked  that  an 
instructional  setting  like  that  used  in  the  OpDemo  might  make  personnel  “now  avoiding  the  vans 
want  to  be  in  the  vans.”  The  implication  here  is  that  training  inside  the  tactical  shelters  (the  vans) 
using  the  embedded  trainers  is  a  somewhat  unpleasant  experience,  one  that  many  AMD 
personnel  avoid  whenever  possible. 

The  four  negative  Hotwash  themes  reported  by  OpDemo  participants  were: 

1.  Frequent  and  recurring  RTOS  “glitches,”  or  technical  problems  experienced  during  the 
Demonstration. 

2.  RTOS  physical  fidelity  and  ergonomic  issues — differences  between  the  RTOS  and  Patriot 
tactical  equipment  or  near- full-physical-fidelity  trainers  such  as  the  PCOFT. 
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3.  RTOS  functional  fidelity  issues — RTOS  functional  limitations  or  differences  between  the 
way  the  system  performs  actions  and  the  way  these  same  functions  currently  are  performed 
using  the  RTOS. 

4.  The  way  ECS-ICC  communications  were  handled  during  the  Demonstration — the  lack  of 
headsets  for  ECS  and  ICC  crews  resulting  in  a  somewhat  noisy  and  chaotic  training  setting. 

Many  of  the  RTOS  features  underlying  these  negative  themes  are  already  being  addressed.  For 
example,  the  RTOS  developer  and  support  contractor,  SAIC,  has  already  responded  to  a  number 
of  the  participants’  comments,  and  others  are  under  investigation  for  correction.  A  summary  of 
RTOS  modifications  currently  perfonned,  underway,  or  under  consideration  as  a  result  of  the 
OpDemo  is  given  in  table  9.  In  all  fairness,  it  should  be  noted  that  the  support  contractor  had 
only  a  two-week  preparatory  period  to  install  and  integrate  the  RTOS  workstations,  develop  and 
test  scenarios,  and  refine  the  training  environment. 

Table  9.  RTOS  issues  and  modifications  resulting  from  participant  hotwash  comments. 


•  First  time  used  as  a  trainer  and  fully  integrated  with  8  consoles 

•  Simulation  running  prototype  software  (TACI/K7/OB) 

•  70  hours  of  training,  13  simulation  aborts  (3  hrs  downtime)  =  96%  available 

•  Fire  Platoon  TAB  54  entries  -  3  instances  (software  fixed) 

•  Fire  Platoon  TAB  01  entries  -  5  instances  (software  fixed) 

•  Battalion  TAB  55  entries  -  2  instances  (software  fixed) 

•  Battalion  TAB  14  entries  -  2  instances  (software  fixed,  being  tested) 

•  One  console  loss  of  power  during  simulation  (moved  power  strip) 

•  Software  Problems  -  18  software  problems  were  identified,  10  were  corrected,  8  under  investigation 

•  Simulation  Differences  from  Tactical  System  -  17  differences  between  the  tactical  system  and  the 
simulation  were  identified  by  the  operators.  1 1  were  corrected  and  6  under  investigation 

•  Requested  Simulation  Enhancements  Identified  by  Operators  and  Trainers 

•  Remote  Tracks  at  ECS 

•  Ability  to  perform  Battalion  Initialization  and  Command  Plan 

•  Ability  to  perform  Mapping 

•  Ability  to  change  TABS  70,  71,  and  72  during  tactical  operations 

•  Ability  to  perform  clutter  map  update 

•  Emulation  of  tactical  communications  networks  between  FPs,  ICC,  and  ADAFCO 

•  FU-FU  capabilities 


4.  Discussion 


As  emphasized  throughout  the  report,  the  RTOS  OpDemo  was  an  exercise  intended  to  illustrate 
and  evaluate  the  feasibility  and  acceptability  of  a  new  training  approach  for  possible  use  in  AMD 
units  using  the  RTOS  as  a  training  vehicle.  The  new  training  approach  emphasized  deliberate 
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practice  within  an  open  instructional  environment  conducive  to  teaching,  coaching,  and 
mentoring. 

It  should  be  noted  that  the  Demonstration  was  not  a  training  experiment.  No  novice  fire  unit 
crews  were  trained  to  criterion  using  the  RTOS  coupled  with  the  new  instruction  instructional 
strategy  and  then  compared  to  a  comparable  control  group.  It  was  more  similar  to  a 
demonstration  and  case  study  than  a  controlled  experiment.  Consequently,  any  generalizations 
of  results  from  the  OpDemo  are  limited  and  must  be  made  cautiously.  However,  despite  these 
limitations  and  cautions,  selected  conclusions  can  be  stated. 

First,  the  OpDemo  results  indicated  that  the  RTOS  has  potential  utility  to  support  AMD  unit 
training  for  air  battle  operations.  Results  from  the  Utility  assessment  strongly  support  this 
conclusion.  In  addition,  results  from  the  Buy-In  survey  indicate  that  participants  viewed  RTOS- 
based  training  as  a  legitimate  training  experience  and  were  willing  to  accept  it  as  a  training  tool. 
Both  of  these  conclusions  are  further  supported  by  participant  responses  to  the  Open-Ended 
survey  questions  and  Hotwash  remarks. 

With  targeted  modifications,  the  results  suggest  that  the  RTOS  has  sufficient  physical  and 
functional  fidelity  to  be  useful  in  the  early  to  mid-ranges  of  the  AMD  training  sequence. 

Potential  categories  of  enhancements  to  the  RTOS  include: 

1 .  Ergonomic  enhancements  to  increase  physical  fidelity. 

2.  Functional  modifications  and  add-ons. 

3.  Instructional  support  features. 

Potential  ergonomic  and  functional  modifications  were  addressed  in  the  previous  section. 
Instructional  support  features  are  simulator  modifications  added  to  enhance  the  instructional 
process  the  device  is  intended  to  support.  These  include  features  such  as  stop-action,  rewind, 
fast-forward,  real-time  replay  from  a  fast-forward  or  rewind  point,  performance  evaluation 
assists,  and  the  like.  It  is  also  essential  that  the  RTOS  support  the  latest  version  of  Patriot  tactical 
software  and  that  software  upgrades  be  accomplished  in  a  timely  manner. 

In  spite  of  the  favorable  review  given  the  RTOS,  AMD  decision  makers  must  bear  in  mind  that 
simulators  by  themselves  do  not  train.  They  are  tools  used  by  good  instructors  operating  within 
a  well-designed  instructional  strategy  to  achieve  training  objectives.  Research  on  training 
effectiveness  has  consistently  shown  that  how  a  simulator  is  used  is  more  important  than  specific 
simulator  features  or  simulation  technologies  (Salas,  Bowers,  &  Rhodenizer,  1998).  Moreover,  a 
large  base  of  training  research  indicates  that  more  fidelity  is  not  always  better  (Alexander, 
Brunye,  Sidman,  &  Weil,  2005).  These  latter  authors  note  that  “a  compromise  must  be  made 
between  physical  fidelity,  related  costs,  and  training  effectiveness,  such  that  an  adequate  match 
can  be  made  between  training  and  real  environmental  elements  and  the  logical  structure  of  [the] 
tasks  [to  be  trained]”  (p.  7).  They  go  on  to  note  that  the  relationships  among  simulation  fidelity, 
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user  acceptance,  and  training  effectiveness  are  not  well  understood.  Each  case  must  be  explored 
heuristic  ally  based  on  its  own  merits,  requirements,  and  knowledge  base.  The  tenn  heuristically 
means  exploration  using  experimental,  trial- and-error  methods.  The  OpDemo  was  an  example  of 
a  heuristic  approach  to  problem  exploration. 

In  spite  of  the  fidelity  conundrum,  the  literature  on  simulators  and  simulations  is  clear  on  one 
point  (Salas,  Bowers,  &  Rhodenizer):  Developers  must  avoid  the  blind  pursuit  of  fidelity  for  its 
own  sake.  These  authors  (p.  200)  note  that  realism  (i.e.,  physical  and  functional  fidelity)  as  a 
sole  simulator  design  criterion  “has  led  us  to  the  point  where  in  the  quest  for  more  realistic 
simulation,  we  have  lost  sight  of  the  true  goal — a  more  effective  training  device  in  tenns  of  both 
training  outcomes  and  cost.  The  two  are  not  the  same.” 

Second,  the  training  strategy  focusing  on  deliberate  practice  and  characterized  by  an  open 
instruction  setting  conducive  to  teaching,  coaching,  and  mentoring  was  well  received  by 
OpDemo  participants.  Buy-In  results,  Open-Ended  survey  results,  and  Hotwash  comments  all 
support  this  conclusion.  Participants  liked  the  instructional  approach  used  in  the  OpDemo  and 
thought  it  should  be  further  applied  in  a  unit  setting.  In  addition,  pre-  and  post-test  results  from 
the  ARM  Classification  training  module  indicate  that  the  approach  was  effective.  Pre-  versus 
post-instruction  test  results  showed  significant  positive  improvement.  Similar  anecdotal  results 
were  observed  for  the  JEZ-MEZ-FEZ  instructional  module. 

Third,  there  was  remarkable  consistency  across  the  categories  of  data  obtained  over  the  course  of 
the  OpDemo.  Utility  results,  Buy-In  Survey  results,  Open-Ended  comments,  and  Hotwash 
remarks  all  paint  a  consistent  picture  of  user  acceptance  and  potential  training  effectiveness. 

Such  consistency  suggests  that  in  spite  of  the  limitations  on  interpretation  and  generalization 
noted  previously,  the  ADA  School  has  a  “green  light”  to  pursue  further  applications  of  the  RTOS 
(or  an  RTOS-like  device)  and  the  modified  instructional  strategies  used  during  the  OpDemo.  In 
this  respect,  the  RTOS  OpDemo  must  be  view  as  a  success.  Moreover,  the  training  set-up  used 
during  the  OpDemo  (RTOS  plus  modified  instructional  approach)  represents  a  partial  prototype 
for  a  solution  to  the  training  deficiencies  that  contributed  to  the  OIF  fratricides  and  that  showed 
up  again  during  the  PDB-6  Limited  User  Test. 

These  latter  two  conclusions  have  considerable  support  within  the  training  effectiveness 
literature.  For  example,  Salas,  Bowers,  and  Rhodenizer  (1998)  argue  that  the  training 
community  must  emphasize  training  systems  that  use  technology  to  promote  learning.  As 
mentioned  previously,  this  requires  a  shift  from  designing  simulation  solely  for  realism — and 
hoping  that  learning  occurs — to  the  design  of  human-centered  training  systems  that  support  the 
acquisition  of  complex  skills.  These  authors  go  on  to  assert  that  achieving  this  goal  will  require 
a  paradigm  shift  from  a  focus  on  simulation  to  a  broader  consideration  of  the  entire  training 
system,  of  which  the  simulator  is  but  one  component.  In  essence,  this  comment  summarizes  the 
approach  used  in  the  OpDemo  and  that  met  with  an  overwhelmingly  favorable  response  from 
participants. 
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In  spite  of  the  success  observed  during  the  OpDemo,  it  would  be  a  mistake  to  deploy  the  RTOS 
or  any  such  device  to  AMD  units  without  an  extensive  training  support  package  and  the  requisite 
command  acceptance,  support,  and  emphasis.  Without  these  dual  elements  of  support,  the 
present  initiative  to  improve  AMD  unit  training  and  enhance  unit  perfonnance  will  fail.  For  the 
RTOS  or  any  similar  simulator,  essential  training  support  requirements  must  include: 

1 .  An  overall  AMD  training  strategy  into  which  such  a  device  fits. 

2.  Enhanced  instructional  methods  (e.g.,  vignettes  supporting  progressive  instructional 
objectives  and  deliberate  practice  versus  simple  free-play  in  a  scenario-based 
environment). 

3.  Pedagogical  modifications  (e.g.,  changes  in  the  way  instructors  conduct  unit  training). 

4.  Courseware  support  (scenario  templates,  training  and  evaluation  guides,  etc.). 

5.  Technical  support  to  the  simulator  itself. 

All  of  these  support  elements  are  critical  to  the  successful  application  an  instructional  regimen 
emphasizing  deliberate  practice  within  an  open  learning  environment,  regardless  of  the  simulator 
selected  for  use. 

Success  in  an  AMD  training  reform  effort  also  may  require  a  re-look  at  the  issue  of  what  must  be 
trained.  Current  job  and  task  analysis  products  for  Patriot  and  other  AMD  systems  may  not 
support  the  instructional  approach  used  during  the  OpDemo  without  extensive  modifications  and 
clarifications.  For  example,  the  JEZ-MEZ-FEZ  and  ARM  Classification  modules  used  during 
the  OpDemo  are  not  included  in  official  Critical  Task  Lists  for  Patriot  MOSs,  yet  these  modules 
involve  skills  that  are  essential  to  successful  job  performance  in  the  contemporary  Patriot 
operational  environment. 
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Appendix  A.  RTOS  Patriot  PDB-5  Tactical  System  Representation 


The  Recon figurable  Tactical  Operations  Simulator  (RTOS)  system  began  its  development  in 
1977  under  contract  with  the  U.S.  Army  Missile  Command  (MICOM).  In  1981,  the 
responsibility  of  the  system  was  transferred  to  the  U.S.  Army  Air  Defense  Artillery  School 
(USAADASCH)  at  Ft.  Bliss,  Texas.  The  RTOS  has  undergone  extensive  enhancements  to 
replicate  the  functions  of  the  deployed  Patriot  Tactical  System  and  Support  simulation  models  of 
other  Air  Defense  systems.  The  U.S.  Army  renamed  the  Patriot  Tactical  Operations  Simulator 
(PTOS)-to-RTOS  in  1989  because  of  the  additional  air  defense  simulations  added,  the  use  of 
recon  figurable  consoles,  and  the  incorporation  of  Distributed  Interactive  Simulation  (DIS)  into 
the  RTOS  architecture.  These  capabilities  enable  the  RTOS  to  be  used  as  a  multiple-system  air 
defense  simulator  with  growth  capabilities  to  incorporate  additional  weapon  or  command-and- 
control  systems  and  participate  in  exercises  and  training  events. 

RTOS  Patriot  simulation  provides  a  high  fidelity  representation  of  the  PATRIOT  software  in  key 
functional  areas  that  support  studies,  exercises  and  training.  Since  its  initial  development,  the 
RTOS  Patriot  model  was  based  on  Data  Processing  System  Requirements  (DPSR)  documents  of 
the  tactical  Patriot  system  and  provides  algorithm  representation  of  the  functional  areas 
supporting  its  role  as  an  analysis  tool  for  the  U.S.  Army’s  Directorate  of  Combat  Developments 
(DCD).  The  RTOS  Patriot  simulator  uses  two  levels  of  modeling:  algorithmic  and  probabilistic. 
During  the  early  development  of  the  Patriot  Missile  System,  the  RTOS  was  used  by 
USAADASCH  to  implement,  verily,  and  validate  the  proposed  Patriot  tactical  software.  Impacts 
on  established  tactics  and  doctrine  were  identified  and  resolved  before  the  tactical  system 
software  became  operational.  A  major  engineering  revision  to  the  tactical  software  could  be 
identified  and  tested  before  implementation  of  the  system  software,  avoiding  costly  and  time- 
consuming  modifications  to  the  tactical  Patriot  software  before  fielding. 

The  RTOS  Patriot  is  capable  of  representing  a  full  battalion  configuration  of  up  to  six  Fire 
Platoons  made  up  of  single  or  dual  console  representations  of  ICC  and  ECSs.  The  RTOS  ECS 
representation  can  either  be  interactive  with  Reconfigurable  consoles  or  they  can  be  represented 
in  batch  (non-interactive)  with  internal  operator  representations. 
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Figure  A-l.  RTOS  Patriot  system  battalion  representation. 

A-l.  RTOS  Users 

The  RTOS  plays  an  important  role  in  the  U.S.  Army’s  participation  in  multilevel  air  defense 
demonstrations  and  exercises.  From  the  earliest  days  of  the  PPO,  through  the  deployment  of  the 
first  operational  Patriot  software  revision  and  refinement  to  the  present  configuration,  the  RTOS 
has  been  an  important  Soldier-in-the-loop  air  defense  tool  supporting  studies  and  exercises. 
Although  the  proponent  and  owner  of  the  RTOS  system  is  USAADASCH,  there  are  other  users 
that  support  the  RTOS.  The  German  Air  Force  (GAF)  used  the  RTOS  system  for  more  than  17 
years.  The  German  Patriot  Project  Office  (GEPO),  located  at  Ft.  Bliss,  Texas,  uses  the  system  in 
conjunction  with  the  Army  to  produce  studies  and  recommendations  for  future  Patriot  system 
changes  as  part  of  bi-lateral  studies.  The  RTOS  Patriot  system  was  procured  by  the  Japan 
Defense  Agency  (JDA)  with  five  RTOS  consoles  in  1996  and  installed  at  Iruma  Air  Base,  Japan 
and  includes  both  PDB-3  and  PDB-4  representations. 
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Figure  A-2.  The  RTOS  system. 

In  addition  to  the  U.S.  Army,  the  German  and  Japanese  Air  Force,  other  users  and  supporters  of 
the  RTOS  include  the  Missile  Defense  Agency  (MDA),  Joint  Tactical  Air  and  Missile  Defense 
Office  (JTAMDO),  Space  and  Missile  Defense  Command  (SMDC),  and  AMRDEC-Software 
Engineering  Directorate  with  permanent  RTOS  systems  located  in  their  facilities  (Ft.  Bliss 
Warfighting  Center,  AMRDEC-SED-Redstone,  JNIC-Colorado  Springs,  JASDF-  Japan,  SMDC 
Battle  Lab  -  Redstone,  and  Virtual  Warfare  Center-St.  Louis). 

A-2.  Model  Configuration 

RTOS  supports  both  Standalone  and  Distributed  simulations.  Stand-alone  simulations  are 
supported  with  the  RTOS  support  programs  that  use  a  scenario  generated  by  the  internal 
Scenario  Generation  Program  supporting  threat  development  of  Air  Breathing  Threats  (ABT), 
Tactical  Ballistic  Missiles  (TBM),  and  Cruise  Missiles.  Distributed  simulations  are  supported  by 
the  RTOS  DIS  capability  which  allows  the  RTOS  to  be  driven  by  other  models  and  interact  with 
other  non-RTOS  models  either  in  the  RTOS  facility  or  externally  over  data/phone  lines.  This 
capability  allows  other  models  to  provide  higher  fidelity  threat  interjection.  The  characteristics 
of  RTOS  are  the  following: 

•  Supports  real-time  execution  on  large  size  scenarios 

•  Supported  by  user-definable  parameters  and  event  scheduling 

•  Supports  customization  via  its  modular  design  to  allow  new  air  defense  models  to  be 
incorporated  into  the  existing  models. 
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RTOS  is  supported  by  Interactive  Support  Programs  that  support  the  users’  ability  to  use  RTOS 
as  a  standalone  trainer  or  as  a  exercise  support  tool.  Support  programs  include:  Scenario 
Generation  Program,  Namelist  Program,  Event  Calendar  File  Program,  Data  Reduction  Program 
and  the  capability  to  supports  individual  console  replay.  The  RTOS  system  allows  the 
Integration  of  Multiple  RTOS  systems  locally  through  Ethernet  connections.  This  capability 
allows  different  RTOS  models  (for  example  Patriot  and  SLAMRAAM)  to  execute  together  using 
the  same  scenario.  Using  the  DIS  standard  and  defined  protocol  data  units,  RTOS  models  can  be 
linked  together  with  other  models  in  other  locations 

A-3.  Hardware 

The  RTOS  system  is  composed  of  off-the-self  hardware  components.  The  simulation  and 
console  computer  systems  are  supported  by  multiple  processor  platforms  to  include  Silicon 
Graphics,  SUN  and  PC.  The  reconfigurable  console  system  provides  a  display  for  the  air 
defense  systems  during  the  simulation  and  is  composed  of  two  30  inch  high  resolution  LCD 
touch  screens  with  replica  keyboards  and  joy-stick.  The  reconfigurable  console  is  modular  and 
easily  transported. 


Figure  A-3.  Latest  RTOS  Reconfigurable  Console  Display. 
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A-4.  Software  Architecture 

The  RTOS  is  a  highly  flexible  and  realistic  simulator  of  multiple  air  defense  weapon  systems. 

Its  design  concept  was  driven  by  the  goal  to  develop  an  analysis  tool  with  an  open  system 
architecture  that  can  be  adapted  to  growing  needs  without  costly  redesign.  The  very  extensive 
software  strictly  follows  actual  system  specifications,  has  been  coded  in  a  high-order  language, 
and  consists  of  over  150,000  lines  of  code.  A  modular  design  structure  allows  for  easy  and  low- 
risk  modifications,  extensions,  and  even  the  addition  of  complete  new  system  models.  This 
flexibility  supports  rapid  and  inexpensive  responses  to  changes  in  weapons  system  software.  It 
also  supports  the  analyst's  need  for  specific  modifications  to  the  simulation  models  necessary  to 
enforce  a  very  particular  system  performance  within  specific  analyses.  The  majority  of  the 
model  areas  of  the  RTOS  are  supported  by  user-modifiable  parameters  and  probabilities.  These 
areas  include  the  following: 

•  Surveillance  and  radar 

•  Missile 

•  Tracking 

•  Identification  Friend  of  Foe  (IFF) 

•  Time  thresholds 

•  Identification 

•  Selection 

•  Data  links 

•  Display. 

A-5.  RTOS  Patriot  Simulation  Software 

The  Patriot  software  of  the  RTOS  is  periodically  upgraded  to  reflect  changes  in  the  fielded 
Patriot  system  and  currently  maintains  baselines  for  PDB-3,  PDB-4,  and  PDB-5  and  initial 
development  of  PDB-6.  The  RTOS  Patriot  ICC  controls  up  to  six  Patriot  Engagement  Control 
Stations  (ECS).  The  reconfigurable  consoles  can  be  configured  as  ICC  and/or  ECS. 

The  Fire  Unit-level  model  in  RTOS  Patriot  is  represented  by  the  major  functions  of: 
Surveillance/Track,  Classification  /  Identification,  Threat  Assessment/Engagement  Decision, 
Weapon  Assignment/Missile  Guidance,  and  Kill  Assessment/ Attacker  Penetration.  The 
Engagement  Decision  and  Weapon  Assignment  (EDWA)  function  is  an  example  of  an 
algorithmic  level  of  modeling,  which  models  the  tactical  operations  processing  for  the  Patriot 
FUs.  The  EDWA  function  provides  threat  evaluation  of  detected  targets,  ID,  target 
classification,  threat  assessment  and  engagement  eligibility  evaluation,  engagement  processing 
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and  response  to  ICC  messages  and  operator  actions.  Processing  to  accomplish  these  tasks  during 
a  scenario  is  based  upon  the  algorithms  outlined  by  the  Patriot  EDWA  DPSR. 

Track  management  at  the  FP  is  an  example  of  a  probabilistic  level  of  modeling.  It  models  the 
detection  of  threat  groups  and  maintenance  of  the  track  position,  velocity  and  selected  track 
status  infonnation.  The  detection  and  tracking  process  is  based  on  a  surveillance  procedure  that 
includes  a  radar  model  of  the  Patriot  FP  radar  characteristics  and  probability  factors  that  can  be 
set  by  the  user.  Masked  terrain  effects  on  the  radar  visibility  are  supported.  High-fidelity 
software  modules,  which  replicate  established  Patriot  data  link,  Patriot  Data  Interface  Fanguage 
(PADIF),  are  employed  to  model  actual  data  communications  behavior. 

The  RTOS  Patriot  model  replicates  the  Command  and  Control  function  of  the  Patriot  battalion 
and  provides  target  evaluation,  engagement-control,  and  processing-command  input  from  the 
operator  or  higher  echelon  (HE).  Evaluation  consists  of  target  Identification  (ID)  and 
Identification  Friend  or  Foe  (IFF)  processing  classification,  threat  assessment  and  protection  of 
assets,  engagement  eligibility,  and  selection  of  a  FP  to  conduct  the  engagement.  Engagement 
control  includes  pre-engagement  processing  of  threats,  computation  of  engagement  data,  and 
maintenance  of  the  to-be-engaged  queue  (TBEQ),  engagement  release  to  subordinate  units,  and 
the  monitoring  and  kill  assessment  of  ongoing  engagements.  Command  input  provides 
processing  of  commands  from  the  operator,  via  switch  action,  or  from  HE  via  data  link  message. 
These  include  engage,  hold-fire,  cease-fire,  and  engage-hold.  Operators  may  release  hold-fire, 
cease-fire,  and  engage-hold  commands.  Major  portions  of  data  communications  between 
subordinate  units  and  the  battalion  C2  element  are  included  in  the  RTOS.  The  communications 
model  uses  PADIF  and  external  tactical  higher  echelon  TADIF-J  data  link.  The  TADIF-J 
datalink  allows  the  RTOS  Patriot  ICC  to  directly  connect  with  any  higher  echelon  element. 

SAIC  has  supported  over  50  simulation  and  training  exercises  with  the  TADIF-J  data  link. 
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Appendix  B.  ARM  Classification  Knowledge  Test 


Date _ 

Soldier  Rank  and  Name: _ 

Unit: _  Crew  Position _  Age: _ 

Military  Education: _ 

Civilian  Education _ 

Do  you  have  a  PC  at  Home? _  Do  you  play  video  games? _ 

PC  or  Console? _ 

1 .  What  are  the  three  tests  that  the  system  uses  to  determine  if  a  track  is  an  ARM 

1. 

2. 

3. 

2.  What  are  some  general  characteristics  that  the  system  uses  to  determine  if  a  track 
is  an  ARM? 


3.  What  does  the  Con  fig  3  Radar  add  as  a  requirement  for  the  system  to  classify  a  track  as 
an  ARM? 

4.  What  is  the  primary  discriminant  for  classifying  a  track? 

5.  Target  classification  is  NOT  reassessed  after  a  classification  is _ . 

6.  The  default  tab  setting  in  Tab  76  page  A  for  range  is _ . 
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7.  The  default  tab  setting  in  Tab  76,  page  A  for  minimum  speed  is _ . 

8.  If  you  have  an  AS-1 1  threat,  your  minimum  speed  tab  setting  is _ as  per  the 

PDB  5.5.2  Speed  Values  for  ARM  Threats  white  paper. 

9.  If  you  have  an  AS- 17a  threat,  your  minimum  speed  tab  setting  is _ as  per  the 

PDB  5.5.2  Speed  Values  for  ARM  Threats  white  paper. 

10.  What  tab  setting  should  you  authorize  to  improve  ARM  detection  performance  as  per  the 
PDB  5.5.2  Improving  ARM  Detection  Performance  White  Paper? 
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Appendix  C.  Utility  Assessment  Survey 


Date _ 

Soldier  Rank  and  Name: _ 

Unit: _  Crew  Position _  Age: _ 

Military  Education: _ 

Civilian  Education _ 

Do  you  have  a  PC  at  Home? _  Do  you  play  video  games? _ 

PC  or  Console? _  What  Genre  (RPG,RTS,FPS): 


Task 

N/S 

P/S 

F/S 

TCA/TDA  Tasks 

1 

441-083-1478  Engage  Targets  on  the  Engagement  Control  Station  (ECS) 

2 

441-083-1479  Evaluate  Preengagement  Data  at  the  Engagement  Control  Station 

3 

441-083-1480  Evaluate  Preengagement  Data  at  the  Information  and  Coordination 
Central 

4 

441-083-1482  Initiate  Target  Engagement  at  the  Information  and  Coordination 

Central 

5 

441-083-1486  Perform  Friendly  Protect  at  the  Engagement  Control  Station  (ECS) 

6 

441-083-1487  Perform  Friendly  Protect  on  the  Information  and  Coordination  Central 
(ICC) 

7 

441-084-1407  Perform  Engagement  Control  Station  (ECS)  Initialization 

TCO/TD  Tasks 

8 

441-EW1-1084  Perform  Engagement  Control  Station  (ECS)  Initialization 

9 

441-EW1-1085  Perform  Engagement  Control  Station  (ECS)  Reinitialization 

10 

441-EW1-1087  Direct  Firing  Battery  System  Reorientation 

11 

441-EW1-1088  Perform  Tactical  Ballistic  Missile  (TBM)  Engagement  Operations 

12 

441-EW1-1089  Perform  Air  Defense  Mission  in  all  Modes  of  Control  for  Engagement 
Control  Station  (ECS) 

13 

441-EW1-1090  Identify  Targets  at  Engagement  Control  Station  (ECS) 

14 

441-EW1-1091  Supervise  Engagement  of  Targets 

15 

441-EW1-1093  Implement  Firing  Doctrine  Changes 

16 

441-EW1-1094  Define  Engagement  Control  Station  (ECS)  Tabular  Displays 

17 

441-EW1-1095  Implement  (ABT)  and  Tactical  Ballistic  Missile  (TBM)  Search  using 
Expanded  Search  Sectors 

18 

441-EW1-1099  Supervise  the  Process  of  Airspace  Control  Orders  (ACO) 

19 

441-EW1-1 1 13  Supervise  Engagement  Operations 

20 

441-EW1-1 121  Analyze  the  Process  of  Evaluation,  Decision,  and  Weapons 

Assignment  (EDWA) 
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21 

441-EW1-1 122  Perform  Friendly  Protection 

22 

441-EW1-1 123  Perform  Duties  and  Responsibilities  as  the  Tactical  Control  Officer 
(TCO) 

23 

441-EW1-1 124  Perform  Air  Breathing  Threat  (ABT)  and  Tactical  Ballistic  Missile 
(TBM)  Search  Using  Expanded  Search  Sectors 

24 

Identify  Mis-Classified  Tracks  that  are  displayed  as  ARM,  TBM,  or  ABT  symbology 

25 

Identify  when  crew  duties  and  responsibility  should  be  modified 

26 

Use  of  a  banded  FEZ ,JEZ,  MEZ  TTPs 

27 

Performing  De-Lousing  TTPs 
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Appendix  D.  RTOS  Demo  Questionnaire 


Primary  MOS 

Duty  MOS 

Rank 

Job/Position 

#  Yrs  Service 

#  Yrs  in  Position 
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Please  comment  on  any  response  of  (1)  Disagree  to  (2)  Agree 


Statement 

Disagree 

(1) 

Agree 

(2) 

1.  The  RTOS  Unit  training  environment  is 
more  conducive  to  coaching,  teaching  and 
mentoring  then  the  current  method  used  to 
train. 

2.  I  learned  new  advanced  Joint  Air  Defense 
Operation  techniques. 

3.  1  learned  techniques  to  evaluate  and  react  to 
off  nominal  situations  (misclassifications, 
anomalies) 

4.  All  things  considered,  this  demo  was  a 
worth  while  training  event. 

5.  I  think  the  RTOS  will  be  a  useful  tool  for 
unit  training. 

6.  Comments:  (Environment,  Training,  RTOS  issues,  General  Impressions). 


7.  What  does  the  RTOS  need  to  do  that  it  does  not  currently  do? 
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Appendix  E.  Open-Ended  Survey  Responses 


SPC/TDA:  RTOS  does  not  have  the  capability  to  surpass  training  on  actual  equipment. 
However,  RTOS  would  be  great  tool  to  deployed  units  where  system  equipment  is  not  readily 
available  for  training.  RTOS  would  be  good  for  use  with  a  tactical  seminar  situation  as  a 
demonstration  tool  for  specific  tactical  situations  and  negative  Patriot  situations  like  fratricide. 

PV2/TDA:  I  think  it  [provides]  a  great  learning  and  teaching  environment.  I  learned  more  here 
in  two  weeks  than  in  most  of  AIT.  It  was  fun  yet  very  productive — good  people,  great  teaching, 
and  good  tools.  Should  be  put  in  all  Patriot  batteries  for  further  teaching  and  learning. 

CW3/TD:  Very  useful  tool  for  S/I  and  Tab  learning.  Useful  for  maintaining  crew  proficiency. 
Good  for  giving  classes  and  showing  what  is  being  taught  to  large  groups.  Another  tool  for 
crews  to  training  and  get  familiar  with  the  system. 

CW3/TD:  Great  training  tool.  Training  with  RTOS  resulted  in  increased  system  knowledge  and 
understanding.  Needs  to  support  training  of  complete  crews — incorporate  25F  and 
c  ommunications . 

SPC/TDA:  Environment  conducive  to  giving  FUs  situation  awareness  (ICC  sit  display  on 
projector).  I  think  it  should  be  at  the  battery  level  for  system  knowledge  and  low  intensity  air 
battles.  Screens  seem  to  get  too  cluttered  for  high  intensity. 

PV2/TDA:  I  liked  it.  I  got  what  the  FU  and  ICC  go  through.  Need  more  cornmo  realism — but 
if  it  did  that  it  wouldn’t  be  as  informational  as  it  is.  Talk  between  FU  and  ICC  in  the  same  room. 

PV2/14E:  It  was  a  well  constructed  program  and  system,  and  I  hope  that  I  can  have  more  work 
with  it.  I  definitely  learned  new  things  and  gained  more  infonnation,  and  I  just  left  the  AIT 
environment.  Need  to  add  fix  or  fight  decision  making  and  FUFU  operations. 

PFC/TDA:  Very  good  training  tool. 

SSG/14E:  Good  and  could  be  a  useful  training  tool  at  the  battalion  or  battery  level  to  training 
soldiers  with  a  different  perspective  than  the  PCOFT  or  in  Van  by  yourself. 

SPC/TCA:  Isolated  training  (the  ARM  Classification  and  JEZ-MEZ-FEZ  modules)  excellent. 
Outstanding  training  system. 

2LT/14E:  Good  training  tool.  Gives  batteries  greater  access  to  hands-on  practice  as  well  as 
ability  to  accentuate  training  lessons  immediately  after  a  verbal  class. 

SPC/14E:  Train  units  at  least  every  other  day  to  make  all  units  battle  ready. 

SPC/14E:  Good  for  training — no  need  to  set  up. 
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SPC/TCA:  Minus  some  minor  system  glitches,  the  system  is  very  useful  and  would  be  a  very 
valuable  asset. 

2LT/TCO:  I  like  it!  It  would  be  better  than  the  PCOFT.  I  think  it  would  be  helpful  at  the  battery 
level. 

SPC/TCA:  Good  system — great  training  tool. 

2LT/TCO:  This  was  excellent  training  because  the  focus  was  on  training  and  not  on  evaluating. 
We  were  allowed  to  ask  as  many  questions  as  we  could  ask  and  there  were  plenty  of  people  to 
give  an  answer.  The  hands-on  is  the  best  method  of  learning,  but  it  is  much  better  when  you 
have  a  presentation  precede  it  the  way  it  was  done  here.  I  hope  to  see  the  RTOS  in  the  near 
future  as  part  of  the  unit’s  equipment.  It  will  facilitate  training  as  often  and  as  simple  as 
possible.  For  once  we  can  focus  on  actual  training  and  not  on  passing  “spinning  wheels”  (an 
1 1th  ADA  Brigade  term  which  means  they  are  doing  an  in  depth  preparation  for  a  certain  event 
usually  a  table  certification).  Our  job  is  air  battles  and  not  mechanics. 

2LT/TCO:  I  believe  the  RTOS  system  is  the  ideal  place  to  teach  additional  skills  and  info 
(initially).  Once  crews  have  been  exposed  to  it,  they  should  refine  it  in  the  van. 

2LT/TCO:  Should  have  somebody  knowledgeable  by  the  TCOs  and  TCAs  position  to  help 
mentor  and  coach  them  during  the  decision  making  process  throughout  the  air  battles.  That  way 
we  can  see  what  we  are  doing  right  and  wrong.  I  think  the  RTOS  is  a  very  complete  program 
and  covers  ah  aspects  of  the  ECS.  A  good  thing  is  that  you  have  the  ICC  right  there  answering 
the  TCO’s  questions. 

2LT/14E:  The  specified  tasks,  scenarios  and  training  prior  to  and  during  RALs  (scenarios)  were 
very  useful.  I  felt  I  had  a  better  understanding  of  my  purpose  and  understand  the  situation, 
threat,  and  capabilities  of  the  system  better  now.  We  need  to  do  some  ACO  (Airspace  Control 
Order)  training. 

PV2/TCA:  I  liked  the  hands-on  training.  I  think  it  would  be  priceless  to  have  one  at  the  battery 
level. 

SSG/14E30:  This  system  would  be  good  training  at  the  platoon  level. 

2LT/TCO:  It  was  good  in  ah  areas.  I  think  it’s  better  than  the  PCOFT. 

2LT/14A:  Good  training.  Helps  before  going  to  the  van. 

SGT/14E:  Great  tool  for  training. 

SPC/14E:  A  few  glitches,  but  is  a  good  training  device. 
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Distribution  List 


ADMNSTR 

DEFNS  TECHL  INFO  CTR 
ATTN  DTIC-OCP  (ELECTRONIC  COPY) 
8725  JOHN  J  KINGMAN  RD  STE  0944 
FT  BEL  VOIR  VA  22060-6218 

DARPA 

ATTN  IXO  S  WELBY 
3701  N  FAIRFAX  DR 
ARLINGTON  VA  22203-1714 

OFC  OF  THE  SECY  OF  DEFNS 
ATTN  ODDRE  (R&AT) 

THE  PENTAGON 
WASHINGTON  DC  20301-3080 

COMMANDING  GENERAL 

US  ARMY  AIR  DEFNS  ARTILLERY  CTR 

AND  FT  BLISS  (5  COPIES) 

ATTN  MGRP  LENNOX 
BLDG  2  SHERIDAN  RD 
FT  BLISS  TX  79916 

ASSISTANT  COMMANDANT 
US  ARMY  AIR  DEFNS  ARTILLERY  CTR 
AND  FT  BLISS  (5  COPIES) 

BLDG  2  SHERIDAN  RD 
FT  BLISS  TX  79916 

COMMANDER 

108TH  ADA  BDE 

ATTN  AFZX-CO  (5  COPIES) 

FT  BRAGG  NC  28310-5100 

COMMANDER 

11TH  ADA  BDE 

ATTN  AFVJ-CO  (5  COPIES) 

FT  BLISS  TX  79916 

COMMANDER 

263RD  AAMDC  (5  COPIES) 

1  NATIONAL  GUARD  RD 
COLUMBIA  SC  29201 

COMMANDER 
3 1ST  ADA  BDE 
ATTN  AFVR-CO  (5  COPIES) 

FT  BLISS  TX  79916 


COMMANDER 

32ND  AAMDC 

ATTN  AFVL-CO  (5  COPIES) 

FT  BLISS  TX  79916 

COMMANDER 
35TH  ADA  BDE 

ATTN  EAAD-H,  PSC3  (5  COPIES) 

APO  AP  96266 

COMMANDER 
69TH  ADA  BDE 

ATTN  AET V -GW -CO,  CMR  475  (5  COPIES) 
PO  BOX  1402 
APO  AE  09036 

COMMANDER 
94TH  AAMDC 

ATTN  AMPC-CG  (5  COPIES) 

MAIL  STOP  577 
FT  SHAFTER  HI  96858 

ARL  HRED  AMEDD  FLD  ELMT 
ATTN  AMSRD-ARL-HR-MM  V  RICE-BERG 
BLDG  40 1 1  RM  2 1 7  1 750  GREELEY  RD 
FT  SAM  HOUSTON  TX  78234-5094 

ARL-HRED  AMCOM  FLD  ELMT 

ATTN  AMSRD-ARL-HR-MO  J  MINNINGER 

BLDG  5400  RM  C-242 

REDSTONE  ARSENAL  AL  35898-7290 

US  ARMY  RSRCH  DEV  AND  ENGRG  CMND 
ARMAMENT  RSRCH  DEV  AND  ENGRG  CTR 
ARMAMENT  ENGRG  AND  TECHNLGY  CTR 
ATTN  AMSRD-AAR-AEF-T 
ATTN  AMSRD-AAR-AEF-T  J  MATTS 
BLDG  305 

ABERDEEN  PROVING  GROUND  MD  21005- 
5001 

ARMY  G1 

ATTN  DAPE-MR  B  KNAPP 
ARMY  G1  MANPRINT  DAPE  MR 
300  ARMY  PENTAGON  RM  2C489 
WASHINGTON  DC  20310-0300 
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ARMY  RSCH  LAB-HRED  JFCOM  JOINT 
EXPERIMENTSATION  J9  JOINT  FUTURES 
LAB 

ATTN  AMSRD-ARL-HR-MJK 
J  HANSBERGER 
115  LAKEVIEW  PARKWAY  STE  B 
SUFFOLK  VA  23435 

ARMY  RSRCH  LAB-HRED 
ATTN  AMSRD-ARL-HR-MU 
M  SINGAPORE 

6501  E  1 1  MILE  RD  MS  284  BLDG  200A  2ND 
FL  RM  2104 

WARREN  MI  48397-5000 

ARMY  RSRCH  LABORATORY-HRED 
ATTN  AMSRD-ARL-HR-MQ 
MR  FLETCHER 

AMSRD-NSC-SS-E  BLDG  3  RM  341 
NATICK  MA  01760-5020 

ARMY  RSRCH  LABORATORY-HRED 
ATTN  AMSRD-ARL-HR-ML  J  MARTIN 
MYER  CENTER  RM  2D3 1 1 
FT  MONMOUTH  NJ  07703-5601 

ARMY  RSRCH  LABORATORY-HRED 
ATTN  AMSRD-ARL-HR-MG  R  SPINE 
BUILDING  333 

PICATINNY  ARSENAL  NJ  07806-5000 

ARMY  RSRCH  LABORATORY-HRED 
ATTN  AMSRD-ARL-HR-MK  J  REINHART 
10125  KINGMAN  RD 
FT  BEL  VOIR  VA  22060-5828 

ARMY  RSRCH  LABORATORY-HRED 
ATTN  AMSRD-ARL-HR-M  M  STRUB 
6359  WALKER  LANE  SUITE  100 
ALEXANDRIA  VA  22310 

ARMY  RSRCH  LABORATORY-HRED 
ATTN  AMSRD-ARL-HR-MN  R  SPENCER 
DCSFDI  HF 

HQ  USASOC  BLDG  E2929 
FT  BRAGG  NC  28310-5000 

ARMY  RSRCH  LABORATORY-HRED 
ATTN  AMSRD-ARL-HR-MW  E  REDDEN 
BLDG  4  RM  332 
FT  BENNING  GA  31905-5400 


ARMY  RSRCH  LABORATORY-HRED 
ATTN  AMSRD-ARL-HR-MT  J  CHEN 
12423  RESEARCH  PARKWAY 
ORLANDO  FL  32826-3276 

ARMY  RSRCH  LABORATORY-HRED 
ATTN  AMSRD-ARL-HR-MD  T  COOK 
BLDG  5400  RM  C242 
REDSTONE  ARSENAL  AL  35898-7290 

ARMY  RSRCH  LABORATORY-HRED 
ATTN  AMSRD-ARL-HR-MH  C  BURNS 
BLDG  1467B  RM  336  THIRD  AVE 
FT  KNOX  KY  40121 

ARMY  RSRCH  LABORATORY-HRED 
ATTN  AMSRD-ARL-HR-MP 
D  UNGV ARSKY 
BATTLE  CMD  BATTLE  LAB 
415  SHERMAN  AVE  UNIT  3 
FT  LEAVENWORTH  KS  66027-2326 

ARMY  RSRCH  LABORATORY-HRED 
ATTN  AMSRD-ARL-HR-MV 
HQUSAOTC  S  MIDDLEBROOKS 
91012  STATION  AVE  RM  1 1 1 
FT  HOOD  TX  76544-5073 

ARMY  RSRCH  LABORATORY-HRED 
ATTN  AMSRD-ARL-HR-MY  M  BARNES 
2520  HEALY  AVE  STE  1 1 72  BLDG  5 1 005 
FT  HUACHUCA  AZ  85613-7069 

ARMY  RSCH  LABORATORY  -  HRED 
AVNC  FIELD  ELEMENT 
ATTN  AMSRD-ARL-HR-MJ  D  DURBIN 
BLDG  4506  (DCD)  RM  107 
FT  RUCKER  AL  36362-5000 

US  ARMY  TRADOC 

BATTLE  LAB  INTEGRATION  &  TECHL 

DIRCTRT 

ATTN  ATCD-B 

10  WHISTLER  LANE 

FT  MONROE  VA  23651-5850 

COMMANDANT  USAADASCH 

ATTN  AMSRD-ARL-HR-ME  A  MARES 

ATTN  ATSA-CD 

5800  CARTER  RD 

FT  BLISS  TX  79916-3802 
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DIRECTOR 

DIRECTORATE  OF  COMBAT 
DEVELOPMENTS 
ATTN  COL  H  L  COHEN 
5800  CARTER  RD 
FT  BLISS  TX  79916-7001 

DIRECTOR 

DIRECTORATE  OF  TRAINING,  DOCTRINE, 
&  LEADER  DEVELOPMENT 
ATTN  COL  R  K  CARL  (5  COPIES) 

ATTN  J  FALLIN 
ATTN  C  WALLET 
2  SHERIDAN  RD,  BLDG  2 
FT  BLISS  TX  79916-7001 

PROJECT  MANAGER 
IAMD  PEO  MS 

ATTN  SFAE-MSLS-SS  R  THOMAS 
PO  BOX  1500 

HUNTSVILLE  AL  35807-3801 

PROJECT  MANAGER 
LTPO  PEO  MS 

ATTN  SFAE-MSLS-LT  E  EDWARDS 
PO  BOX  1500 

HUNTSVILLE  AL  35807-3801 
DIRECTOR 

OFFICE,  CHIEF  OF  AIR  DEFENSE 
ARTILLERY 
ATTN  ATSA-OCADA 
FT  BLISS  TX  79916 

PM  TIMS,  PROFILER  (MMS-P)  AN/TMQ-52 
ATTN  B  GRIFFIES 
BUILDING  563 
FT  MONMOUTH  NJ  07703 

PRODUCT  MANAGER  TIMS 
ATTN  SFAE-IEWS-NS  TIMS 
ATTN  SFAE-IEWS-NS  TIMS  B  GRIFFIES 
BLDG  563 

FT  MONMOUTH  NJ  07703 
DIRECTOR 

QUALITY  ASSURANCE  OFFICE 
ATTN  ATSA-QAO 
FT  BLISS  TX  79916 

SMC/GPA 

2420  VELA  WAY  STE  1866 
EL  SEGUNDO  CA  90245-4659 


TRADOC  CAPABILITY  MANAGER-LOWER 
TIER 

ATTN  COL  R  L  DELGADO 
BLDG  12,  PERSHING  RD 
FT  BLISS  TX  79916-7001 

TRADOC  CAPABILITY  MANAGER-UPPER 
TIER 

ATTN  ATSA-TSM-UT 
FT  BLISS  TX  79916 

US  ARMY  INFO  SYS  ENGRG  CMND 
ATTN  AMSEL-IE-TD  F  JENIA 
FT  HUACHUCA  AZ  85613-5300 

COMMANDER 
US  ARMY  RDECOM 
ATTN  AMSRD-AMR  W  C  MCCORKLE 
5400  FOWLER  RD 

REDSTONE  ARSENAL  AL  35898-5000 

US  ARMY  RSCH  LABORATORY 

ATTN  AMSRD-ARL-CI-OK-TP  S  FOPPIANO 

BLDG  459 

ABERDEEN  PROVING  GROUND  MD  21005 

US  ARMY  RSRCH  LAB 
ATTN  AMSRD-ARL-CI-OK-TP 
TECHL  LIB  T  LANDFRIED 
BLDG  4600 

ABERDEEN  PROVING  GROUND  MD  21005- 
5066 

US  GOVERNMENT  PRINT  OFF 
DEPOSITORY  RECEIVING  SECTION 
ATTN  MAIL  STOP  ID  AD  J  TATE 
732  NORTH  CAPITOL  ST.,  NW 
WASHINGTON  DC  20402 

US  ARMY  RSRCH  LAB 
ATTN  AMSRD-ARL-HR-S  L  PIERCE 
ATTN  AMSRD-ARL-HR-SE  D  HEADLEY 
ATTN  AMSRD-ARL-HR-SE  K  COSENZO 
BLDG  459 

ABERDEEN  PROVING  GROUND  MD  21005 

US  ARMY  RSRCH  LAB 

ATTN  AMSRL-ARL-HR  F  PARAGALLO 

BLDG  459 

ABERDEEN  PROVING  GROUND  MD  21005- 
5066 
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DIRECTOR 

US  ARMY  RSRCH  LAB 

ATTN  AMSRD-ARL-RO-EV  WD  BACH 

POBOX  12211 

RESEARCH  TRIANGLE  PARK  NC  27709 

US  ARMY  RSRCH  LAB 

ATTN  AMSRD-ARL-HR-ME  J  HAWLEY 

FT  BLISS  TX  79916 

US  ARMY  RSRCH  LAB 
ATTN  AMSRD-ARL-D  JM  MILLER 
ATTN  AMSRD-ARL-CI-OK-T 
TECHL  PUB  (2  COPIES) 

ATTN  AMSRD-ARL-CI-OK-TL 
TECHL  LIB  (2  COPIES) 

ATTN  IMNE-ALC-IMS 
MAIL  &  RECORDS  MGMT 
ADELPHI  MD  20783-1197 
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